Businesses often need to monitor the transmission of application and other meaningful data on their networks in real-time. In many cases, it is only possible to monitor such information by monitoring the network itself. This can be the case when there is no way to access application data directly on the host computer or application server. Even where other types of monitoring is possible, network monitoring of application data has advantages over application server, client, and other types of monitoring for the following reasons. First, ease of integration. It is easier to do a “drop in” of new event collection and processing capabilities into an organization when no new software has to be installed on existing devices and new capabilities do not have to be installed on existing servers or integrated into existing server applications. Second, timeliness. The events collected are more timely since they are collected in the network rather than after the application has processed them. This timeliness allows for root cause analysis of performance issues in real-time as well as allowing for detection of otherwise difficult to detect timing problems. Third, completeness. An application problem may be caused by a network problem at any level of the Open Systems Interconnection (“OSI”) Reference Model or stack (i.e., physical, data link, network, transport, session, etc.). Since business applications are being increasingly distributed over multiple servers, each performing a part of the application task, network monitoring becomes increasingly necessary to get a complete picture of the business application's operation. Fourth, localization. Network monitoring makes it easier to locate the source of important events in distributed systems and so identify application/business problems that are caused by network issues. Fifth, application independence. Network monitoring allows collection of data independently from the application. This allows the gathering of data in a way that cannot be compromised and so allows for reliable auditing of application level transactions.
One problem with performing message monitoring of applications via network monitoring is that application and other useful business data such as transaction performance is encapsulated in several layers of communications protocols as described in the OSI Reference Model or stack. For reference, FIG. 1 is a block diagram illustrating the seven-layer OSI Reference Model 100. The physical transmission protocol is typically at the bottom layer (i.e., layer 1), followed by link layer information (i.e., layer 2), then by network information (i.e., layer 3), etc., up to the application data (i.e., layer 7) which may itself consist of multiple levels. In addition, a selection of different protocols may be used at each of these levels. Hence, messages sent at any protocol layer do not stand on their own. Their meaning can only be determined when correlated with other semantic and contextual information such as acknowledgements, time outs, etc., associated with the specific protocol and all protocols lying below it. In addition, deriving meaning out of just the one protocol stack is often not enough. Many applications use multiple application protocols, for example, to transfer initialization information, management information, transactional data, etc. For the application to be fully monitored, the protocol stacks of all these related protocols must be fully decoded and correlated. The problem with existing methods and systems for monitoring application messages to gauge business application performance is that they do not effectively monitor the entire protocol stack of all related application messages and related network problems.
In particular, current methods and systems for monitoring messages interpret and correlate message data based on one of two types of software architectures. The first is a streams based architecture. This architecture relies on entire messages being passed from process to process with each process performing its own unique function. The second is an application programming interface (“API”) based architecture. This architecture relies on data components within messages being passed between software functions via a set of APIs. However, both of these architectures have several disadvantages.
The disadvantages of the streams based architecture are as follows. First, data retention. To lower bandwidth requirements and increase simplicity, streams based systems currently drop lower level protocol data at each protocol level that the data is processed. Modules processing higher level protocol information do not have access to all lower level protocol information related to a message. Also, since there is typically no common storage place for contextual data, the context necessary for each level must be stored by the module responsible for that level. This information is not available to other modules. The result is that only localized decisions can be made as to where to route and how to process based on that layer's information. Second, scheduling. Streams based systems schedule by assigning priorities to messages as they start their flow through the system. Since the context of a given message is not known at this point and the state of the entire system cannot be easily determined, there is no way to prioritize the message based on this context, nor to change the priority of a message as it flows through the system. Third, flexibility. Steams based systems offer some modularity and so allow for some flexibility. However, since each module decodes, encodes, manages memory, etc., as does any full application, the modules are relatively complex even if the job they're doing is relatively small. The result is long development times and code which is often initially unstable. Fourth, processing requirements. Due to the necessity to decode incoming messages and encode outgoing messages within each module, streams based systems have a processing overhead that is above and beyond the actual processing required. If security is required, encryption and decryption must also be included. This extra work requires processing and greatly decreases the performance that may be obtained. Fifth, “downstream” status. With streams based systems, there is no inherent knowledge of the state of the intended destinations. The message is simply routed to various modules until it cannot be sent any further. An error message is then returned. As such, there is no way to “look ahead” and so terminate the message before all the processing has been done. Sixth, latency. Streams based systems have high latency as messages are assembled, transferred, and then disassembled at each hop. There is a large amount of protocol overhead as well to provide reliable data transfer. In addition to this, each message must typically be processed in turn to ensure the state of a message is consistent when a new module starts processing it. This means that messages cannot be processed concurrently, thereby increasing latency.
The disadvantages of the API based architecture are as follows. First, data retention. As with streams based systems, API based systems strip off lower level protocol data at each layer that the data is processed. This information is not available to other layers. The result is that only localized decisions can be made as to where to route and how to process based on that layer's information. Second, scheduling. API based systems typically schedule data on a first come, first served basis as this allows for linear program execution. This works well if all data is of the same priority (e.g., Internet traffic), but quickly falls apart in the presence of integrated services data such as video, voice, and data. Third, flexibility. API based systems generally lack the required dynamic flexibility as the code is “bound” into a particular execution stack at compile time.
A need therefore exists for an improved method and system for monitoring messages passed over a network. Accordingly, a solution that addresses, at least in part, the above and other shortcomings is desired.